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EXAMINER'S AMENDMENT 

1 . An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Benjamin Tabor on 8/14/08. 

Please amend Claims 1, 23, 26, 36, 37, 45, and 47-49 as follows: 

1. (Currently Amended) A method for automating the optimization of search 
results displayed in a search Web page, the method comprising: 

receiving search results provided from a search engine to a user 
according to a search term, wherein the search results are selected and 
ranked by a relevance schema; 

collecting data that represents a performance of each of the 
received search results, the collected data quantifying interactions of 
various users with the search results, the collected data originating from at 
least one of a plurality of various sources having various types of 
valuations , the search results and collected data being initially 
incomparable against each other absent normalization, wherein the 
various sources reflect data of quantified interactions of the various users 
with the search results : 



Application/Control Number: 10/805,731 Page 3 

Art Unit: 2161 

combining the collected data by a normalization procedure that 
generates performance data for each of the search results, wherein the 
performance data reflects behavior of the various users, and wherein the 
normalization procedure comprises: 

(1) aggregating the collected data from the various sources; 

(2) compiling various types of valuations, associated with the 
aggregated data, into common measurements; 

(3) weighting the common measurements based on relative 
importance of the various sources, wherein the relative importance 
porformanco is a reflection of [[the]] a_value of each of the various 
sources as a predictor of relevance of the search results; and 

(4) normalizing the weighted common measurements to 
determine performance data associated with each of the search 
results such that the search results are comparable against each 
other and such that the normalized performance data is 
comparable against expected performance data of the search 
results i nd i cat e d by e xp e ct e d p e rformanc e data ; 

determining whether the expected performance data is below the 
normalized performance data by comparing the normalized performance 
data to the expected performance data for each of the search results; 

when the normalized performance data of a search result of the 
search results is below the expected performance data associated 
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therewith, identifying the search result as underperforming and diagnosing 
the underperforming search result based on results of the comparison 
between the normalized performance data and the expected performance 
data; and 

updating the relevance schema based on the diagnosis such that 
operation of a search engine that provided the search results is adjusted 
to improve relevance of subsequent search results. 

23. (Currently Amended) An automated search result optimization system 
implemented in a computer that provides search results to a user, comprising: 

an input component to receive input data representing a 
performance of a search result generated by a search engine that is 
executed on the computer for a search term, the input data originating 
from a plurality of various sources; 

a data collection processor to collect the input performance data 
from the plurality of various sources wherein the input performance data 
collected includes implicit performance data that describes an aggregation 
of various users interactions during various sessions with the search 
result , wherein the input performance data from one of the plurality of 
various sources is initially incomparable against the input performance 
data from another of the plurality of various sources and the search results 
are initially incomparable against each other absent normalization. 
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wherein the plurality of various sources reflecting collected input 
performance data: 

a comparison processor to compare the collected input 
performance data of the search result to an expected performance data of 
the search result, wherein comparing comprises normalizing input 
performance data by the following steps: 

(1) aggregating the collected input performance data from 
the plurality of various sources: 

(2) compiling various types of valuations, associated with the 
aggregated collected input performance data, into common 
measurements; and 

(3) normalizing the common measurements to determine 
common input performance data associated with each of the 
search results such that search results are comparable against 
each other and such that normalized input performance data of 
search results is comparable against expected performance data of 
the search results: 

a diagnostic processor to determine, based on the comparison, 
whether the collected input performance data diverges from the expected 
performance data bv a quantified threshold, and if so, identify the search 
result as underperforming and diagnose the underperforming search result 
utilizing results of the comparison to select from a set of predefined 
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corrective actions, wherein one of the set of corrective actions is 
implementing an adjustment processor; and 

the adjustment processor to automatically adjust operation of the 
search engine, 

wherein the adjusted operation of the search engine improves the 
performance of the search result by tailoring a ranking of subsequent 
search results in accordance with the collected input performance data, 
and 

wherein an increased weight is attached to the collected input 
performance data from one or more of the various sources based on the 
implicit performance data thereby influencing the automatic adjustment of 
the operation of the search engine. 

26. (Currently Amended) The system of Claim 25, wherein the diagnostic 
processor compares the performance of the search result as represented by the 
normalized collected performance data to the expected performance data of the 
search result, and wherein [[the]] the search result is underperforming when the 
performance is lower than the quantified threshold below the expected 
performance data . 

36. (Currently Amended) The system of Claim_24, wherein the sample test 
data includes data obtained from a test of the search result's relevance 
performed on a sample of a subset of users. 
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37. (Currently Amended) The system of Claim 24, wherein the diagnostic 
processor is further configured to diagnose at least one possible reason why the 
performance of the underperforming search result compares unfavorably against 
the expected performance data , wherein the reason is determined upon 
executing one or more of a set of processes comprising considering whether the 
search result is no longer valid, considering whether the search result appears in 
a poor location, whether a search term that generated the search result is easily 
misspelled, considering whether the search term is too broad to generate a 
meaningful result, or considering whether a search for the search term should be 
constrained to a specific resource. 

45. (Currently Amended) One or more computer-accessible media having 
instructions stored on the media for facilitating the automated optimization of a search 
result in a search result user interface, the instructions comprising: 

receiving search results provided by a search engine in response to 
an inputted search term from a user; 

collecting data that represents a performance of each search result 
of the search results from at least one of a plurality of various sources of 
performance data that includes implicit data collected by quantifying 
interactions of various users during various sessions with each of the 
search results , wherein the performance data from one of the plurality of 
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various sources is initially incomparable against the performance data 
from another of the plurality of various sources and the search results are 
initially incomparable against each other absent normalization, wherein 
the plurality of various sources reflecting the performance data quantifying 
interactions of various users: 

normalizing the collected performance data in accordance with a 
relative importance associated with each of the various sources of the 
performance data, wherein normalizing comprises formatting the 
performance data in a common measurement that is associated with each 
of the search results such that the search results are comparable against 
each other and such that normalized performance data of search results is 
comparable against expected performance data of search results 
assoc i at e d w i th e xp e ct e d p e rformanc e data ; 

comparing the normalized performance data against the expected 
performance data for [fthell each search result; 

based on the comparison, determining whether the normalized 
collected performance data diverges from the expected performance data 
by a quantified threshold and identifying a search result, of the search 
results, as underperforming when it is associated with divergent 
normalized collected performance data; 
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diagnosing at least one possible cause for the underperforming 
search result based on a result of the comparison between the normalized 
performance data and the expected performance data; 

utilizing the at least one possible cause to select from a set of 
predefined corrective actions, wherein one of the set of corrective actions 
is adjusting an operation of the search engine accordance with the 
normalized collected performance data. 

47. (Currently Amended) The computer-accessible media of Claim 45, 
wherein the instruction to normalize the collected performance data that 
describes the user's interactions with the search results in accordance with a 
relative importance of the source of the performance data includes instructions to 
give greater weight to the performance data from the more importance sources of 
the at least one of the plurality of various sources and to combine the 
performance data to reflect the relative importance of the source from which the 
performance data originated. 

48. (Currently Amended) The computer-accessible media of Claim 45, 
wherein the instruction to compare the normalized performance data to the 
expected performance data includes an instruction to determine that the search 
result is underperforming when the normalized performance data is the quantified 
threshold below the expected performance data . 
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49. (Currently Amended) The computer-accessible media of Claim 47, 
wherein the most important source of data is the implicit performance data, and 
the instruction to normalize the collected performance data includes an 
instruction to give implicit performance data greater weight when combining the 
performance data. 



Allowable Subject Matter 

2. The following is an examiner's statement of reasons for allowance: 

In summary, the claimed subject matter found in the independent claims 
regarding user interactions having an effect on the search results by quantifying this 
performance data and the normalization procedures for normalizing this data and 
comparing it against expected performance data of the search results is not reasonably 
taught or made obvious by the prior art(s). 

3. Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 
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Conclusion 

4. Any prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Applicant is advised that, although not used in the rejections 
above, prior art cited on any PTO-892 form and not relied upon is considered materially 
relevant to the applicant's claimed invention and/or portions of the claimed invention. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Brent S. Stace whose telephone number is 571-272- 
8372 and fax number is 571-273-8372. The examiner can normally be reached on M-F 
9am-5:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Apu M. Mofiz can be reached on 571-272-4080. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 

Patent Application Information Retrieval (PAIR) system. Status information for 

published applications may be obtained from either Private PAIR or Public PAIR. 

Status information for unpublished applications is available through Private PAIR only. 

For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 

you have questions on access to the Private PAIR system, contact the Electronic 

Business Center (EBC) at 866-217-9197 (toll-free). 

/Etienne P LeRoux/ /B. S./ 

Primary Examiner, Art Unit 2161 Examiner, Art Unit 2161 



